feat(tracing): per-step obs wrappers inside business Temporal activities - #491

What & why

Two ways to give a business step an obs presence on the Temporal path:

• Option A (tag the ambient span) — don't open an obs span for the step; just stamp the business ids (agentex.business_trace_id / agentex.business_span_id) onto the span the OTel TracingInterceptor already made active (the ambient RunActivity span). Correlation still works, but the step gets no span of its own.
• Per-step wrapper — open a dedicated, step-named obs span, nest that step's downstream hops (httpx / DB) under it, and close it when the step ends. This is the 1:1 mapping the sync path already does.

Before this change, _begin_obs used Option A for any Temporal activity, so every business span in an async turn collapsed onto one obs span, whereas the sync path gives each step its own (1:1). This implements the _in_temporal_activity follow-up called out in #484.

The fix

Option A is only actually required for the SDK's own dispatched start-span / end-span activities (the in_temporal_workflow() path), where start and end run as separate activities that Temporal can route to different workers — so a wrapper opened in start-span could never be closed by end-span (it would leak and its obs_span_id would dangle).

Inside a business activity (an agent turn's own adk.tracing.span, e.g. process_mortgage_turn), start and end run in the same process, so a wrapper is safe there: it nests under the interceptor's ambient RunActivity span and closes in-process.

_begin_obs now discriminates on activity type via _in_tracing_dispatch_activity() (true only for start-span/end-span):

• dispatched tracing activity → tag the ambient span (Option A, unchanged),
• everything else (sync or a business activity) → open a real per-step wrapper.

The now-dead _in_temporal_activity() is removed (its TODO is exactly what this implements). The bounded _OBS_HANDLES registry backstops any mis-discrimination.

Why it matters

With per-step wrappers on the async path you can tell which hop belongs to which step (hops nest under their step, not one giant activity span), each step's persisted obs_span_id points at its own step-named span instead of a shared generic one, and async finally produces the same trace shape as sync.

Span volume on retries

A retried business activity now emits a full per-step wrapper set per attempt, each nested under that attempt's RunActivity (previously they collapsed onto the ambient activity span). This is the intended 1:1 shape, but until the TurnTrace roll-up lands a retried turn surfaces as N per-attempt span sets rather than one PRIMARY + N RETRY view. That roll-up and a multi-replica bounded-_OBS_HANDLES check are tracked as TODO(obs-followup) in trace.py.

Verified on infra-staging (rocket-mock-async-agent), one async turn

Each step (mortgage.advisor.turn, classify_intent, retrieve_docs.kb_query, authz.check.*, tool.*, telemetry.shard.*, synthesis.draft_reply) now gets its own obs span nested under RunActivity, matching the sync path.

Tests

Updated test_temporal_obs_backend.py for the new discriminator, plus two tests asserting: inside a dispatch activity → tag ambient (no wrapper); otherwise → open a wrapper.

🤖 Generated with Claude Code

Greptile Summary

The PR gives business steps inside Temporal activities dedicated observability wrapper spans while retaining ambient-span tagging for the SDK’s separately dispatched tracing activities.

• Adds activity-type discrimination between tracing-dispatch and business activities.
• Prefers the Temporal OTel context when opening wrappers and reading correlation IDs.
• Adds backend-drift diagnostics and focused tests for dispatch and business-activity behavior.

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Begin business span] --> B{Tracing dispatch activity?}
    B -->|Yes| C[Tag ambient Temporal OTel span]
    C --> D[Persist ambient correlation IDs]
    B -->|No| E{Inside any Temporal activity?}
    E -->|Yes| F[Open per-step OTel wrapper]
    E -->|No| G[Open wrapper selected by configured mode]
    F --> H[Persist wrapper correlation IDs]
    G --> H
    H --> I[Register handle]
    I --> J[End business span]
    J --> K[Close and remove wrapper handle]

_{Reviews (8): Last reviewed commit: "refactor(tracing): probe expected backen..." | Re-trigger Greptile}